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1.  Introduction 


Consider  the  “minimum  transform”  ^{y)  of  a  function  F{x)  defined  by 

(1.1)  ^y)  =  min  [F(rr)  -  xy]. 

X 


Then,  under  certain  conditions  on  F,  the  essential  one  being  that  of  convexity, 
the  inverse  relation  is  simply 

(1.2)  F(x)  =  max  [4>(y)  +  xy], 

y 


that  is,  F  is  the  “maximum  transform”  of  f>.  We  shall  refer  to  transforms  of 
either  type  generically  as  “maximum  transforms.” 

In  this  paper  we  shall  show  that  use  of  the  transform  leads  to  a  very  natural 
treatment  of  certain  control  problems. 

The  pair  of  relations  (1.1),  (1.2)  is  strikingly  analogous  in  form  to  a  Fourier 
integral  transformation  and  its  inversion.  The  analogy  is  more  than  fortuitous. 
Consider  a  pair  of  functions  F,  ^  linked  via  the  reciprocal  Fourier  relations 


(1.3) 


exp 

exp 


r^(y)' 

=  const. 

^exp 

~F{x)  -  xyl 

L  c  _ 

J 

c  J 

\^F(xy 

=  const. 

^exp 

’^(y)  +  xy  j 

where  the  integration  contours  are  taken  appropriately,  one  of  them  at  least 
leaving  the  real  axis.  We  suppose  that  c  is  a  small  constant.  As  c  tends  towards 
zero  it  is  plausible  that  the  integrals  in  (1.3)  can  be  asymptotically  evaluated 
by  steepest  descents,  so  that  the  relations  (1.3)  between  F  and  will  reduce  to 
relations  (1.1)  and  (1.2). 

The  significance  of  this  transition  will  emerge  from  our  heuristic  examination 
of  some  rather  special  cases.  We  shall  consider  some  control  problems  in  which 
there  is  a  random  disturbance  whose  variance  is  measured  by  c.  In  some  cases 
these  can  be  solved  by  use  of  Fourier  transforms.  As  c  tends  to  zero  and  the 
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situation  reduces  to  a  deterministic  one,  we  shall  find  that  the  Fourier  treatment 
reduces  to  the  previous  treatment  by  use  of  maximum  transforms. 

The  situation  is,  of  course,  quite  similar  to  the  familiar  transition  between 
wave  optics  and  geometrical  optics,  a  stationary  phase  approximation  in  the 
first  case  leading  to  a  minimum  (or  more  generally  a  stationary)  time  path  in 
the  limit  of  infinitesimally  small  wavelengths.  This  seems  to  be  one  way  in 
which  “natural  extremal  principles”  arise  in  the  physical  world.  However,  in 
the  case  of  the  control  processes,  we  have  an  imposed  extremal  principle:  the 
system  shall  function  optimally  subject  to  given  physical  limitations. 

The  idea  of  the  maximum  transform  was  described  by  Bellman  and  Karush 
([1],  p.  354).  However,  it  has  quite  a  history.  Karlin  treats  the  idea  very 
thoroughly  (under  the  name  of  “conjugate  functions”  [5],  chapter  7),  attributing 
the  work  largely  to  Fenchel  [3].  However,  in  a  work  as  classic  as  Courant  and 
Hilbert’s  Methods  of  Mathematical  Physics  [2]  the  idea  is  quite  well  developed, 
though  not  fully  so  (this  time  under  the  name  of  “involutory  transformations,” 
p.  234)  and  reference  is  made  to  authors  as  early  as  Friedrichs  [4]  and  Trefftz  [6]. 
The  transformation  is,  of  course,  also  related  to  the  classic  Legendre  transfor¬ 
mation. 


2.  Solution  of  linear  problems  by  the  maximum  transform 

Consider  a  linear  control  problem  in  discrete  time  t,  with  state  vector  xt  and 
control  vector  Ui,  so  that 

(2.1)  Xt^i  =  Axt  +  But,  t  integral. 

Typically,  one  might  wish  to  choose  u  so  as  to  minimize  gAuj)  +/(^r)> 

that  is,  so  as  to  minimize  some  integrated  measure  of  control  effort  plus  a 
measure  of  deviation  in  x  from  its  desired  value  at  the  terminal  instant  t  —  T.  If 


(2.2)  F(xi,  T  -  t)  =  min  [  E  gA^i)  +  /(aJr)l, 

Uf  •  -ur-i  L  y  =  4  J 

then  we  have  by  the  dynamic  programming  principle 
p(x,  0)  =  /(x) 


(2.3) 


Define  now 


(2.4) 


F{x,  s)  =  min  [gr-Au)  +  F(Ax  +  Bu,  s  —  1)],  s  =  1,  2, 

U 

4>(y,  s)  =  min  [F{x,  s)  -  y'x], 

X 

<(){y)  =  min  [f{x)  -  y'x], 

X 

ypiz,  t)  =  min  [gt{u)  —  z'u]. 


One  finds  then  readily  from  relations  (2.3),  (2.4)  that 
(2.5)  ^{A'y,  s)  =  4»(y,  s  -  1)  +  H-B'y,  T  -  s). 
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The  first  transformation  in  (2.4)  has  eliminated  the  minimizing  operation  in 
relation  (2.3)  and  left  us  with  a  purely  linear  relation,  (2.5).  If  C  =  exists, 

then  (2.5)  can  be  solved  immediately, 

(2.6)  Hy,  b)  =  Z  T  -  s  +  j)  +  <p{C^y), 

with,  under  appropriate  conditions,  an  inversion 

(2.7)  F{x,  s)  =  max  [4»(^,  s)  +  y's]. 

y 

This  solution  can  be  adapted  also  to  some  cases  in  which  the  values  of  x  and  u 
are  constrained  (to  positive  values,  for  example). 

Suppose  that  the  stopping  rule,  instead  of  being  t  =  T,  is  a  time  independent 
one,  X  G  D.  If  the  loss  functions  g  are  also  time  independent,  then  one  can 
expect  the  same  of  F,  so  that  instead  of  (2.3),  one  has 

(2.8)  F{x)  prescribed  by  /(x),  {x  G  D), 

(2.9)  F(x)  =  min  [g{u)  +  F{Ax  +  Bu)],  {x  G  25). 

U 

If  now 

(2.10)  4>(y)  =  min  [F(x)  —  y'x], 

then  on  a  “negligible  overshoot”  assumption,  we  derive  an  approximate  relation 
from  (2.9) 

(2.11)  Hy)  -  HCy)  +  H-B'Cy). 

By  “negligible  overshoot,”  we  mean  that  XigV  implies  that  Xt+i  =  Axt  +  But 
lies  in  or  only  just  out  of  ID.  Relation  (2.11)  does  not  in  itself  determine  4>  or  F, 
and  must  be  supplemented  by  the  boundary  condition  (2.8). 

All  these  relations  have  obvious  continuous  time  analogues.  If 

(2.12)  X  =  ^  =  ox  +  jSw, 

(2.13)  F{xt,  T  —  t)  =  min  \_f^^  Oriur)  dr  +/(x7’)], 

then  we  find  that  corresponding  to  (2.5)  and  (2.11)  we  have  the  relations 

(2.14)  +  = 

(2.15)  V'A-^  =  f(-fi'y), 

where  d^/dy  is  the  column  vector  of  first  order  derivatives.  We  have  written 

(2.15) ,  the  analogue  of  (2.11),  as  an  equality  rather  than  as  an  approximation 
because  there  will  presumably  be  no  overshoot  into  D  in  case  (2.12).  The  point 
requires  proof,  of  course,  and  raises  issues  such  as  the  continuity  of  F  at  the 
boundary  of  D,  the  stopping  region. 
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As  is  usual  in  dynamic  programming  problems,  we  have  put  all  the  emphasis 
on  the  determination  of  the  loss  function  F.  This  is  a  sufficient  preliminary, 
and  generally  seems  to  be  a  necessary  one,  to  the  immediate  practical  problem 
of  determining  the  policy  or  control  function  u{x). 

3.  Some  examples 

Suppose  that  the  state  variable  re  is  a  scalar  for  which 

(3.1)  X  =  ax  u, 

and  that  we  wish  to  minimize  /  (X  +  /iw®)  dt  under  a  stopping  rule  x  =  Xq.  That 
is,  we  wish  to  choose  w  in  (3.1)  so  as  to  bring  x  to  the  value  xo  in  such  a  way  as 
to  minimize  the  integral  of  X  +  fiu^ — a  criterion  that  gives  weight  both  to  time 
taken  and  the  amount  of  “control  energy”  employed. 

If  the  future  loss  at  (re,  t)  is  F(x,  t),  then  the  dynamic  programming  equation  is 

(3.2)  Ft  +  min  [X  +  tiu^  +  {ax  +  u)F^  =  0, 

U 

with  boundary  condition 

(3.3)  F(rro,  t)  =  0. 

The  subscripts  in  (3.2)  denote  partial  derivatives.  Now,  in  fact,  F  will  be  in¬ 
dependent  of  t,  so  that  (3.2)  reduces  to 

(3.4)  X  +  axF^ 

(3.5) 

Equation  (3.4)  can  in  fact  be  solved  subject  to  condition  (3.3);  we  find 

(3.6)  F{x)  -  H{x)  -  H{xo), 
where 

(3.7)  ff(x)  =  ^  {.y*  +  sgn  (X  -  X,)[X(X^  +  1)>'*  +  sinh-‘  A-]}, 

(3.8)  X  =  xa(;>/X)>'“. 

However,  in  a  situation  with  more  variables  the  equation  analogous  to  the  non¬ 
linear  relation  (3.4)  would  have  been  quite  intractable.  Even  in  the  present  case 
there  is  an  indeterminacy  in  the  solution  of  (3.4)  (leading  to  the  sgn  (X  —  Xo) 
term  in  the  solution)  which  must  be  resolved  by  special  arguments. 

Consider  now  the  use  of  the  minimum  transform  of  F.  This  obeys  an  equation 
(see  (2.15)) 

ay^'iy)  =  ^  ^  2/^ 


4m 


FI  =  0, 


1 


(3.9) 
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of  a  simple  linear  form,  which  can  be  integrated  immediately  to  give 

(3.10)  ^y)  =  const.  +  ^logy  - 

Applying  the  inversion  (1.2)  to  (3.9),  we  recover  solution  (3.6).  In  this  case 
F{x)  can  be  evaluated  explicitly  by  either  method,  but  in  general  it  is  probably 
true  that  the  most  explicit  solution  for  F  would  be  just  the  representation  (1.2), 
with  4>  a  solution  of  the  linear  equation  (2.15). 

A  rather  more  interesting  problem  is  the  following  one  in  which  control  is 
restricted  by  imposing  a  limit  rather  than  a  cost  on  “fuel.” 

Suppose  that  the  state  variables  are  scalars  x,  w  with 

(3.11)  X  =  u, 

(3.12)  w  =  -u\ 

and  that  the  only  cost  is  a  terminal  cost  B,t  t  =  T  of  vx^  —  6w.  The  control  u 
is  to  be  chosen  to  minimize  this  terminal  cost  subject  to  the  restriction  ^  0 
for  t  ^  T.  That  is,  w  has  the  interpretation  of  a  “fuel  reserve,”  and  one  wishes 
to  achieve  x{T)  —  0  as  closely  as  possible,  but  residual  fuel  at  f  =  T  still  has 
a  value  6.  The  case  is  rather  more  realistic  when,  instead  of  (3.11),  we  have 

(3.13)  X  =  u, 

and  it  can  be  reduced  to  the  present  one  by  invariance  arguments. 

Let  s  =  T  —  t  denote  “time  to  go”  and  let  the  future  loss  at  x,  w,  s  be 
F(x,  w,  s).  We  have  then 


(3.14) 

F{x,  w,  0)  =  vx^  —  6w, 

(3.15) 

Ft  =  min  \uFx  — 

s  >  0, 

so  that 

(3.16) 

Fa=  0, 

ti;  =  0,  s  >  0, 

(3.17) 

II 

>  0,  s  >  0. 

From  (3.16)  we  find  that 

(3.18)  F{x,  0,  s)  =  px^. 

The  nonlinear  equation  (3.17)  can  in  fact  be  solved  subject  to  boundary  condi¬ 
tions  (3.14)  and  (3.18).  However,  use  of  the  minimum  transform  again  gives 
a  much  more  expeditious  treatment.  Define 

(3.19)  ^{y,  z,  s)  =  min  [F(x,  w,  s)  —  sy  +  wz'\. 

X 

w  ^0 

We  find,  by  the  same  derivation  as  that  of  equation  (2.15),  that 

(3.20)  =  M(y,  z), 
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(3.21)  M 


if  the  minimizing  w  value  in  (3.19)  is  zero, 
if  the  minimizing  w  value  in  (3.19)  is  positive. 


The  y,  z  values  corresponding  to  it?  =  0  as  minimizing  value  in  (3.19)  are  those 
maximizing  ^  xy  for  some  s,  that  is,  those  for  which  =  0.  We  thus  have 
either 

(3.22)  =  0,  $.  =  0, 

or 

(3.23)  ^  =  0,  ^^>0. 


We  have  further  from  (3.14)  and  (3.19), 

(3.24)  Hy,  0)  = 

L  — 00,  z  <  6. 


For  minimal  4>  we  choose  the  option  (3.23)  rather  than  (3.21)  for  s  >  0,  and 
so  in  general 


(3.25) 


z^e‘ 
z  <e. 


The  fact  that  =  —  oo  {or  z  <  6  implies  that  the  marginal  utility  of  fuel  is 
never  less  than  0,  as  one  would  expect. 

We  shall  have  then 


(3.26) 


F{x,  w,  s)  =  max  [$  +  xy  —  wz'\ 


=  max  [-^  +  !)  +  xy  -  wz]- 

g  ^9 


2  Uj 


The  maximizing  z  value  must  certainly  be  positive,  and  so  given  by 

(3.27) 

if  this,  quantity  exceeds  6.  Thus, 

(3.28)  F  =  max  ~  \y\{wsy'^  +  xy~^) 

if  the  maximizing  y  value  in  (3.28)  makes  expression  (3.27)  exceed  6;  otherwise, 
z  =  6  and 


(3.29) 


F  =  max[-g-g  +  X!/-w«_. 
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Now  expression  (3.28)  has  its  maximum  value  at 

(2v[x  —  if  X  ^  (wsY^^, 

(3.30)  2/  =  j  0  if  la: I  ^  (wsY'^, 

[2i'[a:  +  (lys)*^^]  if  a:  ^  —(wsY^^, 

so  that  the  expression  is  equal  to  v[x  —  sgn  (a:)(i«s)^''^]^  if  jxj  >  (wsY^^,  zero 
otherwise.  However,  the  condition  |^|  >  2d{s/wy'^  implies  that  one  of  the  solu¬ 
tions  (3.30)  can  be  accepted  only  if 

(3.31)  v\x  —  sgn  {x){wsy^\{s/wy'^  ^  6 
or 

(3.32)  la:l  ^  (wsy'^  -f-  ^  {w/sy^] 


otherwise,  F  is  determined  by  (3.29).  We  thus  obtain  the  final  solution 


(3.33) 


fv[x  —  sgn  (a:)(wjs)^''^]^  if  ja:!  ^  (wsy^^  {d/v){w/sy'^, 


U— ; - TT  —  6w  otherwise. 

LI  -f  vs/d 


These  two  very  different  functions  can  both  be  verified  to  be  solutions  of  (3.17), 
but  the  approach  taken  here  is  both  simpler  and  more  automatic  than  direct 
argument  from  (3.17).  The  two  regions  are  respectively  those  of  “fuel  shortage” 
and  “fuel  plenty”;  we  find  that 


(3.34) 


■-|sgn  ix){w/sy^ 
vx 

6  -b  vs 


in  the  two  cases,  respectively. 

A  point  to  be  noted  is  that,  whereas  the  loss  function  F  takes  different  analytic 
forms  in  different  regions,  its  transform  #  does  not.  It  is  in  the  inversion  from 
that  one  discovers  the  existence  of  qualitatively  different  regimes. 


4.  A  stochastic  control  process 

Consider  again  the  process  associated  with  equation  (3.1),  but  let  this  now  be 
modified  to 

(4.1)  X  =  ax  -f  w  -b  e, 

where  e  is  a  “white  noise”  process  with  zero  mean,  whose  integral  per  unit  time 
has  variance  c.  Such  a  process  is  a  perfectly  proper  one  when  understood  in  the 
context  of  generalized  processes. 

Relations  (3.2)  and  (3.4)  now  become  modified  to 

Ft  +  min  [X  -b  nu^  +  {ax  -b  u)F/]  +  |  =  0, 

ti  " 


(4.2) 
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(4.3)  \  +  caF,-^Fl  +  ^F„  =  Q. 

If  we  take 

(4.4)  j(,)  =  exp[-^] 

as  independent  variable  instead  of  F,  then  equation  (4.3)  takes  the  linear  form 

(4.5)  c^Jxx  +  2acxJx  --  J  =  0. 

M 

This  is  just  the  Hermite  equation,  whose  general  solution  is  a  linear  combination 
of  two  integrals  of  the  form 

(4.6)  j  I 

where  the  contour  of  integration  follows  the  imaginary  ^-axis,  with  an  indenta¬ 
tion  either  to  the  right  or  the  left  of  the  origin,  and  4>(?/)  is  the  function  defined 
by  equation  (3.10).  These  two  integrals  are  to  be  combined  in  such  a  way  that 
F(x)  is  zero  and  minimal  at  a:  =  Xq.  Now,  as  c  tends  to  zero  (and  the  stochastic 
relation  (4.1)  degenerates  to  the  deterministic  one  (3.1),  it  is  plausible  that  the 
integrals  (4.6)  can  be  evaluated  by  steepest  descents;  and  we  see  then  that  the 
integral  relation  between  F  and  ^  determined  by  relations  (4.4)  and  (4.6)  will 
reduce  to  the  maximum  transformation  (1.2).  That  is,  the  maximum  transforma¬ 
tion  (and  its  inverse)  arises  as  a  limiting  case  of  a  Fourier  transformation  and 
its  inverse,  corresponding  to  a  “minimum  phase”  evaluation  in  a  transition  from 
wave  to  geometric  optics. 

However,  the  picture  is  not  always  this  simple;  for  example,  it  does  not  seem 
possible  to  linearize  the  stochastic  version  of  equation  (3.22), 

(4.7)  ^  +  I 

by  a  simple  transformation  F.  In  such  cases  the  stochastic  problem  is  presumably 
not  soluble  in  terms  of  linear  transforms,  and  the  maximum  transform,  suc¬ 
cessful  in  the  deterministic  limit,  must  be  the  limit  version  of  some  more  general 
transformation. 


5.  The  replacement  of  minimization  by  a  functional  integration 

Suppose  one  has  the  problem  of  calculating  a  loss  function 
(5.1)  F{x,  t)  =  min  L[x(t),  x{t),  t]  dr, 

x(t) 

where  x  =  x(t),  and  the  integrated  loss  is  minimized  with  respect  to  x(t),  for 
t  <  T  ^  T.  One  might  replace  the  analytically  awkward  minimization  by  a 
functional  integration;  that  is,  suppose  that  F  was  instead  given  by 
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(5.2)  exp  (—i^/7)  =  const.  J  f  d^{T)  exp  ^  — L  dr/y^. 

Here  by  the  right  member  we  mean  the  functional  integral 

(5.3)  lim  j  dxi  j  dx^  •  •  •  j  dxn  exp  S  L  ^  Xj,  t  + 

where  5  =  (T  —  t)/N  and  x^  =  x.  Expression  (5.3)  may  well  involve  terms  in  5 
which  lead  to  singularities  as  5  — ^  0.  These  terms  are  absorbed  in  the  “constant” 
in  (5.2);  the  point  is  that  even  if  expression  (5.3)  does  not  tend  to  a  positive 
limit  as  5  — >  0,  the  quotient  of  two  such  expressions  for  different  x  may  well  do 
so,  and  this  is  all  that  is  needed. 

As  7  tends  to  zero  in  (5.2),  one  will  expect  that  the  value  of  F  given  by  relation 
(5.2)  will  become  identical  with  that  given  by  (5.1). 

Suppose  one  is  dealing  with  a  stochastic  dynamic  programming  problem,  so 
that  integral  (5.1)  must  be  averaged  in  some  way.  In  the  modification  (5.2)  we 
can  combine  this  averaging  with  the  functional  integration,  and  so  have  the 
great  advantage  that  both  our  operations,  integration  and  averaging,  are  linear. 

The  validity  of  the  replacement  of  evaluation  (5.1)  by  (5.2)  (or  its  limit  as 
7  — »  0)  is  not  clear,  still  less  are  the  real  life  implications  of  replacing  an  averaging 
of  i  L  dr  by  that  of  exp  (—7“^  /  Ldr).  Nevertheless,  we  shall  find  that  an 
application  of  this  technique  to  the  two  examples  we  have  discussed  leads  to 
explicit  and  suggestive  results,  and  this  may  justify  some  investigation  of  the 
method. 

Consider  the  problem  of  section  4,  the  minimization  of  /  (X  +  nu^)  dr  subject 
to  (2.1)  and  the  stopping  rule  x  =  a:o.  If 

(5.4)  J(x)  = 
we  have 

(5.5)  J{x)  =  const.  E  /.../ 

duir)  exp  [-;/  (X  +  nu^)  drj, 

SO  that,  to  within  o{8)  terms, 

(5.6)  J(,x)  =  j  j  [x'  -  X  -  S(ca  +  «)]* 

-  i  (X + 

Taking  Fourier  transforms 

(5.7)  K{^)  =  j  e^^J{x)dx, 
we  find  that  (5.6)  becomes 

(5.8)  K({)  =  m  -  «8f)  exp  (|  +  ^)]  +  o(8), 


or 
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and 


(5.10)  log  K(^)  =  const.  “  ~  ^  ~  (|  +  ^) 

From  (5.7),  (5.10), 


(5.11)  J(x)  =  /  e-“fK(^) 

jiirt  y  —  00 

=  const,  j  exp{  -  y-^[\l^{y)  +  xy]}  dy/y, 

where  y  =  iy^  and 

(5.12)  =  + 

Note  the  close  resemblance  between  '^{y)  and  the  minimum  transform  ^{y) 
of  (3.10),  also  between  (5.11)  and  (4.6).  In  the  deterministic  case  c  =  0,  we 
can  go  to  the  limit  7-^0  and  obtain  the  correct  evaluation  of  F{x)  as  the  max¬ 
imum  transform  of  '^{y).  The  apparent  effect  of  a  stochastic  element  t  is  to 
decrease  y  io  tky/{y  -{■  ^yc);  it  is  plain  that  the  value  of  7  must  be  related  to 
that  of  c  if  sensible  results  are  to  be  obtained. 

Consider  the  stochastic  version  of  the  second  problem  associated  with  equa¬ 
tions  (3.11)  and  (3.12) 


(5.13) 


X  =  U 
W  =  —u^, 


the  only  cost  being  the  terminal  one  px^  —  dw  Sit  t  =  T,  or  s  =  {T  —  t)  ==  0. 
Defining 

(5.14)  J{x,  w,  s)  = 

(5.15)  K(|,  r],  s)  =  dx  jj  dw  e^^'^J{x,  w,  s), 

we  find,  using  the  same  functional  integration  technique  as  before,  that 

(5.16)  rj,  s)  =  const,  {r]  -  e/y)-^  |exp  “  2  ®]}’ 

so  that 

(5.17)  J  =  =  const,  jj  d^dr} 

=  const.  11  (z  -  «)-■  exp [-{(;  +  J  +  y) 

+  xy  —  dz  dw, 


where  y  =  iy^,  z  =  777. 
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The  square  bracketed  expression  in  the  exponent  is  just  the  bracketed  expres¬ 
sion  in  relation  (3.26),  at  least  in  the  deterministic  case  c  =  0.  The  fact  that 
maximization  is  restricted  to  z  ^  0  in  (3.26)  now  corresponds  to  the  fact  that 
in  the  asymptotic  evaluation  (7  0)  of  integral  (5.17)  one  must  consider  not 

only  the  saddle  points  of  the  exponent,  but  also  the  pole  at  z  =  6. 
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